[LEADS-504] Simplify Ragas context precision metric names#280
[LEADS-504] Simplify Ragas context precision metric names#280bsatapat-jpg wants to merge 1 commit into
Conversation
|
Warning Review limit reached
Next review available in: 55 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (9)
WalkthroughUpdates Ragas context metric naming across implementation, validation, configuration, examples, and documentation. Adds deprecated-name compatibility for old metric identifiers, while consolidating context precision into ChangesContext precision metric consolidation
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/lightspeed_evaluation/core/metrics/ragas.py (1)
221-243: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winMissing
turn_data is Noneguard, unlike sibling_evaluate_context_precision.
_evaluate_context_precision(added right below in this same diff) and_evaluate_context_recallboth short-circuit with a clear error whenturn_data is None._evaluate_context_utilizationdoesn't, so aNoneturn_datasilently falls through_extract_turn_datato empty query/response/contexts and triggers a real LLM call with empty inputs instead of a clean, immediate error message.🩹 Suggested fix for consistency
if is_conversation: return None, "Context utilization is a turn-level metric" + if turn_data is None: + return None, "TurnData is required for context utilization" + query, response, contexts = self._extract_turn_data(turn_data)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lightspeed_evaluation/core/metrics/ragas.py` around lines 221 - 243, Add the same explicit turn_data None guard to _evaluate_context_utilization that _evaluate_context_precision and _evaluate_context_recall use, so it fails fast with the same clear error instead of calling _extract_turn_data and ContextUtilization.score with empty inputs. Keep the behavior consistent with the sibling metric helpers in ragas.py by checking turn_data before unpacking query/response/contexts and returning the same immediate error path.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/lightspeed_evaluation/core/metrics/ragas.py`:
- Around line 221-243: Add the same explicit turn_data None guard to
_evaluate_context_utilization that _evaluate_context_precision and
_evaluate_context_recall use, so it fails fast with the same clear error instead
of calling _extract_turn_data and ContextUtilization.score with empty inputs.
Keep the behavior consistent with the sibling metric helpers in ragas.py by
checking turn_data before unpacking query/response/contexts and returning the
same immediate error path.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: ebb2c3bc-950c-499a-ac25-09b5748c8302
📒 Files selected for processing (5)
config/evaluation_data.yamlconfig/system.yamlexamples/02_metrics/context_quality/system.yamlsrc/lightspeed_evaluation/core/metrics/ragas.pysrc/lightspeed_evaluation/core/system/validator.py
xmican10
left a comment
There was a problem hiding this comment.
LGTM, thanks! Minor nit, please update the documentation so it is in sync with the updates.
asamal4
left a comment
There was a problem hiding this comment.
This is not just a rename change - we need to have backward compatibility with deprecation notice in the code..
Old long names continue to work as aliases (log a deprecation warning)
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/lightspeed_evaluation/core/system/validator.py (1)
468-472: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDuplicate canonicalization logic.
The
canonical = DEPRECATED_METRIC_NAMES.get(metric, metric)+ dual-membership check is repeated verbatim for turn-level and conversation-level validation. Consider extracting a small helper, e.g._is_metric_known(metric, known_metrics), to avoid drift if the logic changes later.♻️ Proposed refactor
+ `@staticmethod` + def _is_metric_known(metric: str, known_metrics: set[str]) -> bool: + """Check if a metric (or its canonical alias) is in the known set.""" + canonical = DEPRECATED_METRIC_NAMES.get(metric, metric) + return metric in known_metrics or canonical in known_metrics + def _validate_metrics_availability(self, data: EvaluationData) -> None: ... for turn_data in data.turns: if turn_data.turn_metrics: for metric in turn_data.turn_metrics: - canonical = DEPRECATED_METRIC_NAMES.get(metric, metric) - if ( - metric not in self._turn_level_metrics - and canonical not in self._turn_level_metrics - ): + if not self._is_metric_known(metric, self._turn_level_metrics): turn_data.add_invalid_metric(metric) ...Also applies to: 482-486
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lightspeed_evaluation/core/system/validator.py` around lines 468 - 472, The turn-level and conversation-level metric checks in validator.py duplicate the same canonicalization and membership logic. Extract that repeated logic into a small helper such as _is_metric_known(metric, known_metrics) in Validator, and have both validation branches call it instead of inlining DEPRECATED_METRIC_NAMES.get(...) and the dual-membership check. Keep the helper generic so both _turn_level_metrics and the conversation-level known metrics use the same source of truth and future changes stay consistent.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/lightspeed_evaluation/core/system/validator.py`:
- Around line 468-472: The turn-level and conversation-level metric checks in
validator.py duplicate the same canonicalization and membership logic. Extract
that repeated logic into a small helper such as _is_metric_known(metric,
known_metrics) in Validator, and have both validation branches call it instead
of inlining DEPRECATED_METRIC_NAMES.get(...) and the dual-membership check. Keep
the helper generic so both _turn_level_metrics and the conversation-level known
metrics use the same source of truth and future changes stay consistent.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 1aa56e71-37ca-480e-aa50-d7cbba2cff0a
📒 Files selected for processing (9)
README.mdconfig/evaluation_data.yamlconfig/system.yamldocs/EVALUATION_GUIDE.mdexamples/02_metrics/context_quality/README.mdexamples/02_metrics/context_quality/system.yamlsrc/lightspeed_evaluation/core/constants.pysrc/lightspeed_evaluation/core/metrics/ragas.pysrc/lightspeed_evaluation/core/system/validator.py
✅ Files skipped from review due to trivial changes (2)
- README.md
- examples/02_metrics/context_quality/system.yaml
🚧 Files skipped from review as they are similar to previous changes (3)
- config/evaluation_data.yaml
- src/lightspeed_evaluation/core/metrics/ragas.py
- config/system.yaml
Description
Type of change
Tools used to create PR
Identify any AI code assistants used in this PR (for transparency and review context)
Related Tickets & Documents
Checklist before requesting a review
Testing
Summary by CodeRabbit